Skip to main content

Guide: Set up AWS

This guide helps you set up your AWS account on DataStori. 🚀

DataStori runs data pipelines in your environment using AWS Fargate and securely integrates with your account using a cross-account IAM role.

One-click deployment with CloudFormation​

You can set up AWS in one click with DataStori's CloudFormation template. The stack creates the resources DataStori needs in your AWS account and sends their details back to DataStori, so you do not copy ARNs or IDs by hand.

Before you start​

  • Select the correct region. Check the region selector at the top right of the AWS console, and switch to the region where you want your data and pipelines before you create the stack. Every resource is created in the region the stack runs in.
  • Have a VPC and subnets ready. You select an existing VPC and subnets when you launch the stack. DataStori's CloudFormation template does not create a VPC or subnets. The subnets need outbound internet access so pipeline tasks can pull their container image and call your source APIs.

Launch the stack​

  1. In the DataStori portal, open Infrastructure, choose AWS, and click Launch Stack under Quick Setup via CloudFormation. The AWS console opens a stack named DataStori-Connect with the template and your connection details filled in.
  2. Confirm the region at the top right of the AWS console, as described above.
  3. Leave DataStoriToken and ExternalCallbackUrl unchanged.
  4. Under Networking, select your VpcId and SubnetIds.
  5. Set AssignPublicIp to match those subnets: ENABLED for public subnets that route to an Internet Gateway, or DISABLED for private subnets that route through a NAT Gateway. If it does not match, pipeline tasks cannot pull their image and will fail to start.
  6. Acknowledge that the stack creates IAM resources, then click Create stack.
  7. Return to the portal. The Infrastructure page updates when the stack finishes.

The launch link is valid for one hour. If it expires, click Launch Stack again.

Resources the template creates​

ResourcePurpose
S3 bucketStores pipeline output. Named datastori-<account-id>-<id>, encrypted, with public access blocked. Kept if you delete the stack.
Security groupAttached to pipeline tasks in your selected VPC. Allows outbound traffic only.
ECS clusterRuns pipeline tasks on AWS Fargate.
Task execution roleLets Fargate pull the container image and write logs. It has no access to your data.
Task roleUsed by the pipeline code. It can read and write only the bucket above.
Management policyLets DataStori run, stop, and inspect ECS tasks, register task definitions, and pass the two task roles to ECS.
Cross-account roleTrusts DataStori's AWS account (595301649076) and carries the management policy.
Callback Lambda function and roleSends the created resource details back to DataStori when the stack is created.

The stack fills in the values listed in the summary below, apart from the VPC and subnets you select.

Adjust resources or enter values manually​

You can still open the AWS Infrastructure form after the stack finishes, or instead of using it. Click Enter manually instead on the Infrastructure page. Use the form to adjust the sizing of your pipeline tasks with Fargate CPU and Fargate Memory (MB), or to provide the values yourself.

Summary​

DataStori uses the following values to run pipelines in your AWS account. The CloudFormation stack creates or fills them in for you, and you can view or edit them in the AWS Infrastructure form.

  1. Your AWS Account ID: 123456789012
  2. VPC ID: vpc-0123abcd
  3. Subnet IDs: subnet-abcde123, subnet-fghij456
  4. Assign Public IP: ENABLED for public subnets, or DISABLED for private subnets
  5. Security Group IDs: sg-5678efgh
  6. S3 Bucket Name: datastori-123456789012-abcd1234
  7. S3 Bucket Region: us-east-1
  8. ECS Cluster ARN: arn:aws:ecs:region:account-id:cluster/YourClusterName
  9. Task Execution Role ARN: arn:aws:iam::account-id:role/<task-execution-role-name>
  10. Task Role ARN: arn:aws:iam::account-id:role/<task-role-name>
  11. DataStori Cross-Account Role ARN: arn:aws:iam::account-id:role/<cross-account-role-name>
  12. CloudWatch Log Group ARN (Optional): If you have a preferred logging destination.